The Dog That Didn’t Bark... Interpreting Non Significance

Hypothesis testing is a crucial component of science. This leads to guidelines (often ignored) in most discipline including psychology. Unfortunately, most focus on significant effects. Nonsignificant effects are sidelined, in spite of their importance to scientific progress. This study reports a survey of practicing scientist on how they would report and interpret explicit scenarios with non-significant effects. There was no consensus on interpretation in terms of predicting future results. Respondents agreed about how to report the significance of a hypothesis test. Most chose not to report any descriptive statistics, or the sample size, or anything about power, or sufficient information to enable replication or meta-. These results shed light on statistical thinking and so should enable more useable guidelines. For nonsignificant effects, the importance of a priori power is emphasised. “It is impossible to prove a negative, i.e. non-existence of some phenomenon”. Furthermore, “falsifiability”, the ability to prove a positive, is claimed to be the very hallmark of science. This perception of the nature of science may be one reason that the search for statistically significant results appears to dominate empirical research guidelines. Examples of such guidelines include: (APA, 2001; Nickerson, 2000; Roberts & Pashler, 2000; Steingrimsson & Luce, 2005; Wilkinson, 1999)for psychology; (Duran et al., 2006) for education and (Campbell, Elbourne, & Altman, 2004; CONSORT, 2001; QUORUM, 2000; STROBE, 2005)for medicine. Most guidelines focus on significant results only, and recommend reporting effect sizes. Only Nickerson(, 2000) explicitly recommends giving a priori power for non-significant results. It is our contention that identifying an absence of difference is one of the most important tools of science, both theoretically and practically. In the physical sciences, conservation laws (e.g. energy, speed of light ion a vacuum) are cornerstones of theory. For psychologist, it is important to establish an absence of difference in the following situations: 1. when theory predicts equality constraints (Birnbaum, 2004; Steingrimsson & Luce, 2005); 2. when establishing lack of evidence for harm of beneficial procedure, e.g. the MMR vaccine or side effects of drugs; 3. when establishing that an effect is highly improbably, e.g. psycho kinesis; and 4. when establishing that controls such as counterbalancing have been successful. Power is crucial for all these applications. The current study reports an internet survey on the views of practicing scientists on how to report and interpret non-significant results. The aim is a better understanding researchers’ perceptions in order to underpin guidelines on non-significance. There is continuing evidence that many guidelines are ignored (Cumming et al., 2007).